Aggregated Estimators and Empirical Complexity for Least Square Regression

Numerous empirical results have shown that combining regression procedures can be a very efficient method. This work provides PAC bounds for the L2 generalization error of such methods. The interest of these bounds are twofold. First, it gives for any aggregating procedure a bound for the expected risk depending on the empirical risk and the empirical complexity measured by the Kullback-Leibler divergence between the aggregating distribution ρ̂ and a prior distribution π and by the empirical mean of the variance of the regression functions under the probability ρ̂. Secondly, by structural risk minimization, we derive an aggregating procedure which takes advantage of the unknown properties of the best mixture f̃ : when the best convex combination f̃ of d regression functions belongs to the d initial functions (i.e. when combining does not make the bias decrease), the convergence rate is of order (log d)/N . In the worst case, our combining procedure achieves a convergence rate of order p (log d)/N which is known to be optimal in a uniform sense when d > √ N (see [10, 15]). As in AdaBoost, our aggregating distribution tends to favor functions which disagree with the mixture on mispredicted points. Our algorithm is tested on artificial classification data (which have been also used for testing other boosting methods, such as AdaBoost).